Sonic Shapes : Visualizing Vocal Expression

Sound has been an overlooked modality in visualization. Why? Because it is ephemeral. We experience it as it happens, often in community with others. Then, the sound is gone. Furthermore, sound in human communication is multidimensional and includes the semantic meaning of words, the meaning of expressive verbal gestures (paralingual and prosodic components), the nonvocal gestures, and relational gestures. Even though we are able to record sound and play it back, we typically focus more on the semantic meaning of words. In this paper, we describe a voice analytics toolkit and present visualizations that focus on the relational and expressive verbal gestures in speech. By making the overlooked channels of human communication visible and persistent, we make it possible to see beneath the surface of our words. This insight will potentially enable the development of new applications for speech therapy, the quantization and visualization of vocal trends common to speakers with medical conditions such as autism spectrum disorder (ASD), the characterization and visualization of communication patterns common in different kinds of relationships and cultures, and the development of new kinds of creative, multimodal works.